(19) 



J 



(12) 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(11) 



EP 1 286 544 A1 



EUROPEAN PATENT APPLICATION 

published in accordance with Art. 158(3) EPC 



(43) 


Date of publication: 


(51) 


mt CI 7: H04N 5/93, G1 1 B 20/10 




26.02.2003 Bulletin 2003/09 








(86) 


International application number: 


(21) 


Application number: 01921965.8 


PCT/JP01/03416 


(22) 


Date of filing: 20.04.2001 


(87) 


International publication number: 








WO 01/082609 (01 .1 1 .2001 Gazette 2001/44) 


(84) 


Designated Contracting States: 


(72) 


Inventors: 




AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


• 


HAMADA, Toshiya, SONY CORPORATION 




MC NL PT SE TR 




Shinagawa-ku, Tokyo 141-0001 (JP) 




Designated Extension States: 


• 


KATO, Motoki, SONY CORPORATION 




AL LT LV MK RO SI 




Shinagawa-ku, Tokyo 141-0001 (JP) 


(30) 


Priority: 21.04.2000 JP 2000185479 


(74) 


Representative: Pratt, Richard Wilson et al 








D. Young & Co, 


(71) 


Applicant: Sony Corporation 




21 New Fetter Lane 




Tokyo 141-0001 (JP) 




London EC4A 1 DA (GB) 



(54) RECORDING APPARATUS AND METHOD, REPRODUCING APPARATUS AND METHOD, 
RECORDED MEDIUM, AND PROGRAM 



(57) From input moving picture data, a thumbnail 
picture representing the moving picture data is generat- 
ed as first thumbnail data. Form the moving picture data, 
a thumbnail picture of a picture extracted as being a 
characteristic picture, or a thumbnail picture of a picture 



specified by a user, is generated as second thumbnail 
data, and the first and second thumbnail data are re- 
corded as respective independent groups on the record- 
ing medium. So, the user is allowed to select recorded 
data readily using the recorded picture data. 
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Description 

Technical Field 

[0001] This invention relates to a recording method 
and apparatus, a reproducing method and apparatus, a 
recording medium and a program, in which a thumbnail, 
representative of a picture, is appended to data. 

Background Art 

[0002] Recently, a variety of types of optical discs 
have been proposed as a recording medium that can be 
removed from a recording apparatus. These recordable 
optical discs have been proposed as a large capacity 
medium of several GBs and are thought to be promising 
as a medium for recording AV (audio visual) signals, 
such as video signals. Among the sources (supply 
sources) of digital AV signal, to be recorded on this re- 
cordable optical disc, there are CS digital satellite broad- 
cast and BS digital broadcast. Additionally, the ground 
wave television broadcast of the digital system has also 
been proposed for future use. 

[0003] The digital video signals, supplied from these 
sources, are routinely compressed under the MPEG 
(Moving Picture Experts Group) 2 system. In a recording 
apparatus, a recording rate proper to the apparatus is 
set. If digital video signals of the digital broadcast are 
recorded in the conventional image storage mediums 
for domestic use, digital video signals are first decoded 
and subsequently bandwidth-limited for recording. In 
the case of the digital recording system, including, of 
course, the MPEG1 Video, MPEG2 video and DV sys- 
tem, digital video signals are first decoded and subse- 
quently re-encoded in accordance with for the recording 
rate and the encoding system proper to the apparatus 
for subsequent recording. 

[0004] However, this recording system, in which the 
supplied bitstream is decoded once and subsequently 
bandwidth-limited and re-encoded prior to recording, 
suffers from the deteriorated picture quality. If, in record- 
ing compressed digital signals, the transmission rate of 
input digital signals is less than the recording rate for 
the recording and/or reproducing apparatus, the method 
of directly recording the supplied bitstream without de- 
coding or re-encoding suffers from deterioration in the 
picture quality to the least extent. However, if the trans- 
mission rate of the input digital signals exceeds the re- 
cording rate of the recording and/or reproducing appa- 
ratus, it is indeed necessary to re-encode the bitstream 
and to record the so re-encoded bitstream, so that, after 
decoding in the recording and/or reproducing appara- 
tus, the transmission rate will be not higher than the up- 
per limit of the disc recording rate. 
[0005] If the bitstream is transmitted in a variable rate 
system in which the bit rate of the input digital signal is 
increased or decreased with time, the capacity of the 
recording medium can be exploited less wastefully with 



a disc recording apparatus adapted for transiently stor- 
ing data in a buffer and for recording the data in a burst 
fashion than with a tape recording system having a fixed 
recording rate imposed by the fixed rpm of the rotary 
5 head. 

[0006] Thus, it may be predicted that, in the near fu- 
ture when the digital broadcast is to become the main- 
stream, an increasing demand will be raised for a re- 
cording and/or reproducing apparatus in which broad- 

10 cast signals are recorded as digital signals, without de- 
coding or re-encoding, as in a data streamer, and in 
which a disc is used as a recording medium. 
[0007] In reproducing a recording medium, having re- 
corded thereon plural data, such as data of program 

15 made up e.g., of video or speech data, using the above- 
described apparatus, there is presented a problem that, 
with an increasing volume of data recorded on the re- 
cording medium, the pre-playback processing, such as 
processing as to which program is to be reproduced or 

20 from which scene the program is to be reproduced, be- 
comes complicated. 

Disclosure of the Invention 

25 [0008] It is an object of the present invention to enable 
data for reproduction to be selected easily by adding a 
thumbnail representative of the data to the data to be 
reproduced. 

[0009] A recording method according to the present 
30 invention a recording apparatus including means for 
generating, from input moving picture data, a thumbnail 
picture representing the moving picture data, as first 
thumbnail data, and for generating, from the moving pic- 
ture data, a thumbnail picture of a picture extracted as 
35 a characteristic picture, or a thumbnail picture of a pic- 
ture specified by a user, as a second thumbnail data, 
and 

means for recording the first thumbnail data and the sec- 
ond thumbnail data, generated by the generating means 
40 as independent groups on a recording medium. 

[0010] Preferably, the generating means generates 
the first and second thumbnail data as respective inde- 
pendent files. 

[0011] Preferably, the generating means generates 
45 the first and second thumbnail data as respective inde- 
pendent data blocks and as one file. 
[0012] Preferably, the generating means generates 
first management data supervising the first thumbnail 
data with a number corresponding to the first thumbnail 
so data. The generating means also generates second 
management data supervising the second thumbnail 
data with a number corresponding to the second thumb- 
nail data, whilst the recording means records the first 
and second management data in the recording medium . 
55 [0013] Preferably, the first and second management 
data include data specifying the format of the picture da- 
ta of the thumbnail data being supervised. 
[0014] Preferably, the recording means records pic- 
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ture data of the thumbnail picture contained in the first 
or second management data in terms of a block of a 
preset size as a unit. 

[0015] Preferably, the recording means records the 
information representing the referencing destination of 
the first thumbnail data as a separate file on the record- 
ing medium. 

[0016] Preferably, the recording means further 
records the information indicating the referencing des- 
tination of the thumbnail picture contained in the second 
thumbnail data. 

[0017] A recording method according to the present 
invention includes a generating step of generating, from 
input moving picture data, a thumbnail picture repre- 
senting the moving picture data, as f irst thumbnail data, 
and for generating, from the moving picture data, a 
thumbnail picture of a picture extracted as a character- 
istic picture, or a thumbnail picture of a picture specified 
by a user, as a second thumbnail data, and a recording 
controlling step of exercising control for recording the 
first thumbnail data and the second thumbnail data, gen- 
erated by the generating step as independent groups 
on a recording medium. 

[0018] A program for a recording medium according 
to the present invention includes a generating step of 
generating, from input moving picture data, a thumbnail 
picture representing the moving picture data, as first 
thumbnail data, and for generating, from the moving pic- 
ture data, a thumbnail picture of a picture extracted as 
a characteristic picture, or a thumbnail picture of a pic- 
ture specified by a user, as a second thumbnail data, 
and 

a recording controlling step of exercising control for re- 
cording the first thumbnail data and the second thumb- 
nail data, generated by the generating step as inde- 
pendent groups on a recording medium. 
[0019] A program according to the present invention 
has a computer execute a generating step of generat- 
ing, from input moving picture data, a thumbnail picture 
representing the moving picture data, as first thumbnail 
data, and for generating, from the moving picture data, 
a thumbnail picture of a picture extracted as a charac- 
teristic picture, or a thumbnail picture of a picture spec- 
ified by a user, as a second thumbnail data, and a re- 
cording controlling step of recording the first thumbnail 
data and the second thumbnail data, generated by the 
generating step as independent groups on a recording 
medium. 

[0020] A reproducing apparatus according to the 
present invention includes first readout means for read- 
ing out management data supervising picture data of a 
thumbnail picture representing the contents of the pic- 
ture data in case playback of the picture data is com- 
manded, second readout means for reading out the pic- 
ture data based on the management data read out by 
the readout means, third readout means for reading out 
picture data of a thumbnail picture of a picture extracted 
as a characteristic picture from the picture data or pic- 



ture data of a thumbnail picture of a picture specified by 
a user, and management data supervising the picture 
data, and fourth readout means for reading out the pic- 
ture data based on the management data read out from 

5 the third readout means. 

[0021] A reproducing method according to the 
present invention includes a first readout controlling 
step of reading out management data supervising pic- 
ture data of a thumbnail picture representing the con- 

10 tents of the picture data in case playback of the picture 
data is commanded, a second readout controlling step 
of controlling the readout of the picture data based on 
the management data readout-controlled by the 
processing of the first readout control means, a third re- 

15 adout controlling step of reading out picture data of a 
thumbnail picture of a picture extracted as a character- 
istic picture from the picture data or picture data of a 
thumbnail picture of a picture specified by a user, and 
management data supervising the picture data, and a 

20 fourth readout controlling step of reading out the picture 
data based on the management data readout-controlled 
by the processing of the third readout control step. 
[0022] A program for a recording medium according 
to the present invention includes a first readout control- 

25 |jng step of controlling the readout of management data 
supervising picture data of a thumbnail picture repre- 
senting the contents of the picture data in case playback 
of the picture data is commanded, a second readout 
controlling step of controlling the readout of the picture 

30 data based on the management data readout-controlled 
by the processing of the f irst readout control means, a 
third readout controlling step of reading out picture data 
of a thumbnail picture of a picture extracted as a char- 
acteristic picture from the picture data or picture data of 

35 a thumbnail picture of a picture specified by a user, and 
management data supervising the picture data, and a 
fourth readout controlling step of reading out the picture 
data based on the management data readout-controlled 
by the processing of the third readout controlling step. 

40 [0023] A program according to the present invention 
includes a first readout controlling step of controlling the 
readout of management data supervising picture data 
of a thumbnail picture representing the contents of the 
picture data in case playback of the picture data is com- 

45 manded, a second readout controlling step of controlling 
the readout of the picture data based on the manage- 
ment data readout-controlled by the processing of the 
first readout control means, a third readout controlling 
step of reading out picture data of a thumbnail picture 

50 of a picture extracted as a characteristic picture from the 
picture data or picture data of a thumbnail picture of a 
picture specified by a user, and management data su- 
pervising the picture data, and a fourth readout control- 
ling step of reading out the picture data based on the 

55 management data readout-controlled by the processing 
of the third readout control step. 
[0024] A recording medium according to the present 
invention has, recorded thereon, first data comprised of 
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picture data, thumbnail picture data of a thumbnail pic- 
ture extracted from the picture data, Indicating the con- 
tents of the picture data, and management data for su- 
pervising data of the thumbnail picture, and second data 
comprised of thumbnail picture data of a thumbnail pic- 
ture of a picture extracted as a characteristic picture 
from the first stated picture data or picture data of a 
thumbnail picture of a picture specified by the user and 
management data supervising the picture data. 
[0025] With the recording method and apparatus and 
the first program of the present invention, a thumbnail 
picture representative of input moving picture data is 
generated from the input moving picture data, as a first 
thumbnail data, whilst a thumbnail picture of a picture 
extracted as being a characteristic picture from the mov- 
ing picture data or a thumbnail picture of a picture spec- 
ified by the user is generated as second thumbnail data. 
The first and second thumbnail data are recorded as 
separate independent groups on the recording medium. 
[0026] With the reproducing method and apparatus 
and the second program according to the present inven- 
tion, if playback of an AV stream is commanded, man- 
agement data supervising picture data of a thumbnail 
picture representing the contents of the AV stream is 
read out, whilst picture data is read out based on the 
management data read out by readout means. From the 
AV stream, picture data of a thumbnail picture of a pic- 
ture extracted from it as a characteristic picture, or pic- 
ture data of a thumbnail picture of a picture specified by 
a user, and management data supervising the picture 
data, are read out, and the picture data are read out 
based on the read-out management data. 
[0027] Other objects, features and advantages of the 
present invention will become more apparent from read- 
ing the embodiments of the present invention as shown 
in the drawings. 

Brief Description of the Drawings 
[0028] 

Fig.1 shows a configuration of a recording and/or 
reproducing apparatus according to the present in- 
vention. 

Fig.2 shows the relation between Clip and PlayList. 
Fig.3 is a UML doagram of a structure supervising 
an AV stream.. 

Fig. 4 shows a directory structure of a DVR system.. 

Fig.5 shows info.dvr. 

Fig.6 illustrates DVRVolume(). 

Fig. 7 illustrates UIApplnfovolume(). 

Fig.8 illustrates TableOf PLayl_ist(). 

Fig.9 illustrates UlApplnfo PlayListQ. 

Fig. 10 illustrates zzzzz.clip. 

Fig. 11 illustrates ClipMark(). 

Fig.12 illustrates Mark. 

Fig. 13 illustrates xxxxx.rpls and yyyyy.vpls. 

Fig. 14 illustrates PlayListQ. 



Fig. 15 illustrates Playltem(). 

Fig. 16 illustrates Playl_istMark(). 

Fig. 17 illustrates a menu thumbnail. 

Fig. 18 illustrates a mark affixed to PlayList. 
5 Fig.l9 illustrates a mark affixed to the PlayList. 

Fig.20 illustrates a file in which to store a thumbnail. 

Fig.21 illustrates menu_thmb/mark.thmb. 

Fig. 22 illustrates ThumbnailQ. 

Fig. 23 illustrates thumbnail_picture_format. 
10 Figs.24A and 24B illustrate a method for storing 

thumbnail picture data in tn_block(). 

Fig. 25 illustrates another directory file structure. 

Fig. 26 illustrates menu_thmb/mark.thmb associat- 
ed with the file structure shown in Fig.25. 
is Fig.27 illustrates TableOfplayList() associated with 

the file structure shown in Fig.25. 

Fig.28 illustrates PlayList() associated with the file 

structure shown in Fig.25. 

Fig.29 illustrates UlApplnfo PlayList() associated 
20 with the file structure shown in Fig.25. 

Fig.30 is a flowchart showing the sequence for cre- 
ating a mark thumbnail. 

Fig. 31 is a flowchart showing the sequence for cre- 
ating a menu thumbnail. 
25 Fig.32 shows another directory structure of the DVR 
system. 

Fig. 33 shows the syntax of the thumbnail header 
information 

Fig.34 illustrates display_aspect_ratio. 
30 Fig. 35 illustrates color_space. 

Fig.36 shows the syntax of a thumbnail picture data 
file. 

Fig.37 illustrates data storage to tn_block. 
Fig.38 illustrates a medium. 

35 

Best Mode for Carrying out the Invention 

[0029] Referring to the drawings, a recording method 
and apparatus, a reproducing method and apparatus, a 

40 recording medium and a program, according to the 
present invention, will be explained in detail. Fig. 1 
shows a typical illustrative structure of a recording and/ 
or reproducing apparatus 1 embodying the present in- 
vention. First, the portion of the structure responsible for 

45 recording externally input signals on a recording medi- 
um is explained. The recording and/or reproducing ap- 
paratus 1 is able to be fed with and record analog or 
digital data. 

[0030] Analog video signals and analog audio signals 
so are fed to terminals 11,12, respectively. The video sig- 
nals input to the terminal 11 are output to an analysis 
unit 1 4 and to an AV encoder 1 5. The audio signals input 
to the terminal 1 2 are output to the analysis unit 1 4 and 
to the AV encoder 15. The audio signals fed to the ter- 
55 minal 1 2 are output only to the AV encoder 1 5. The anal- 
ysis unit 14 extracts feature points, such as scene 
changes, from the input video signals. 
[0031] The AV encoder 15 encodes input video and 
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audio signal to output the system information (S), such 
as an encoded video stream (V), an encoded audio 
stream (A) and AV synchronization, to a multiplexer 16. 
[0032] The encoded video stream is a video stream 
encoded e.g., with the MPEG (Moving Picture Expert 
Group) 2 system, whilst the encoded audio stream is an 
audio stream encoded in accordance with the MPEG1 
system. The encoded audio stream is e.g., an audio 
stream encoded in e.g., the MPEG1 system or an audio 
stream encoded in accordance with the Dolby AC3 
(trademark) system. The multiplexer 16 multiplexes the 
input video and audio streams, based on the input sys- 
tem information, to output a multiplexed stream through 
a switch 1 7 to a multiplexed stream analysis unit 1 8 and 
to a source packetizer 19. 

[0033] The multiplexed stream is e.g., an MPEG-2 
transport stream or an MPEG2 program stream. The 
source packetizer 19 encodes the input multiplexed 
stream into an AV stream composed of source packets 
in accordance with an application format of a recording 
medium 100 on which to record the stream. The AV 
stream is processed in ECC (error correction and cod- 
ing) unit 20 and a modulation unit 21 with appendage of 
ECC codes and with modulation, before being output to 
a write unit 22, which then writes (records) an AV stream 
file based on a control signal output by the controller 23. 
[0034] The transport stream , such as digital television 
broadcast, input from a digital interface or a digital tele- 
vision tuner, is input to a terminal 13There are two re- 
cording systems for recording the transport stream, in- 
put to the terminal 1 3, one being a transparent recording 
system and the other being a system in which recording 
is preceded by re-encoding aimed at e.g., lowering the 
recording bit rate. The recording system designating in- 
formation is input from a terminal 24 as a user interface 
to a controller 23. 

[0035] In the transparent recording of the input trans- 
port stream, a transport stream, input to a terminal 13, 
is output to a multiplexed stream analysis unit 1 8 and to 
the source packetizer 1 9. The ensuing processing of re- 
cording an AV stream on a recording medium is the 
same as that of encoding and recording input audio and 
video signals, as described above, and hence is not ex- 
plained here for simplicity. 

[0036] If the input transport stream is re-encoded and 
subsequently recorded, the transport stream, input to 
the terminal 13, is input to a demultiplexer 26. The de- 
multiplexer 26 demultiplexes the input transport stream 
to extract the video stream (V), the audio stream (A) and 
the system information (S). 

[0037] Of the stream (information) extracted by the 
demultiplexer 26, the video stream is output to the AV 
decoder 27, while the audio stream and the system in- 
formation are output to the multiplexer 16. The AV de- 
coder 27 decodes the input video stream to output the 
reproduced video stream to the AV encoder 15, which 
then output the encoded video stream V to the multi- 
plexer 16. 



[0038] On the other hand, the audio stream and the 
system information, output from the demultiplexer 26 
and input to the multiplexer 16, are multiplexed, based 
on the input system information, and output as a multi- 

5 plexed stream to the multiplexed stream analysis unit 
18 and to the source packetizer 19 through switch 17. 
The ensuing processing until recording of the AV stream 
on the recording medium 100 is similar to that of encod- 
ing and recording the video signals and hence is not ex- 

10 plained specifically. 

[0039] The recording and/or reproducing apparatus 1 
of the present embodiment records a file of the AV 
stream on the recording medium 1 00, while also record- 
ing the application database information which accounts 

15 for the file. The application database information is pre- 
pared by the controller 23. The input information to the 
controller 23 is the feature information for the moving 
picture from the analysis unit 14, the feature information 
of the AV stream from the multiplexed stream analysis 

20 unit 1 8 and the user command information input at a ter- 
minal 24. 

[0040] The feature information of the moving picture, 
supplied from the analysis unit 1 4, is the information per- 
tinent to a picture characteristic of the input moving pic- 

25 ture signals, and may, for example, include the specify- 
ing information (marks), such as a program start point, 
a scene change point or start and end points of the com- 
mercial (CM), or the thumbnail picture information of a 
picture of the specified site. 

30 [0041] The feature information of the AV stream from 
the multiplexed stream analysis unit 18 is the informa- 
tion pertinent to the encoding information of the AV 
stream for recording, such as the address information 
of an l-picture in an AV stream, encoding parameters of 

35 the AV stream, transition point information of the encod- 
ing parameters in the AV stream, orthe information per- 
tinent to characteristic pictures in a video stream 
(marks). 

[0042] The specifying information of the user from ter- 
40 minal 24 is the information in the AV stream, such as the 
specifying information of the playback domain as spec- 
ified by the user, character letters which account for the 
contents of the playback domain or the bookmarks or 
resuming points for the user to set his or her favorite 
45 scene. 

[0043] Based on the aforementioned input informa- 
tion, the controller 23 creates a database of the AV 
stream (Clip), a database of a group (PlayList) of play- 
back domains (Playltem) of the AV stream, manage- 

50 ment information of the recorded contents of the record- 
ing medium 1 00 (info.dvr) and the information on thumb- 
nail pictures. Similarly to the AV stream, the application 
database information, constructed from the above infor- 
mation, is processed in the ECC unit 20 and the modu- 

55 lation unit 21 and input to the write unit 22, which then 
records a database file on the recording medium 100. 
[0044] The above-described application database in- 
formation will be explained subsequently in detail. 
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[0045] When the AV stream file recorded on the re- 
cording medium 100 (files of picture data and speech 
data) and the application database information, thus re- 
corded on the recording medium 100, are reproduced, 
the controller 23 first commands a readout unit 28 to s 
read out the application database information from the 
recording medium 100. The readout unit 28 reads out 
the application database information from the recording 
medium 100, which then reads out the application da- 
tabase information from the recording medium 100 to 
send the application database information through de- 
modulation and error correction processing by a demod- 
ulating unit 29 and an ECC decoder 30 to the controller 
23. 

[0046] Based on the application database informa- 
tion, the controller 23 outputs a list of PlayList recorded 
on the recording medium 100 to a user interface of the 
terminal 24. The user selects the PlayList, desired to be 
reproduced, from the list of Play Lists. The information 
pertinent to PlayList, specified to be reproduced, is input 
to the controller 23. The controller 23 commands the re- 
adout unit 28 to read out the AV stream file necessary 
in reproducing the PlayList. In accordance with the com- 
mand, the readout unit 28 reads out the corresponding 
AV stream from the recording medium 1 00 to output the 
read-out AV stream to the demodulating unit 29. The AV 
stream, thus input to the demodulating unit 29, is de- 
modulated by preset processing and output through the 
processing by the ECC decoder 30 to the source pack- 
etizer 1 9. 

[0047] The source depacketizer 31 converts the AV 
stream of the application format, read out from the re- 
cording medium 100 and processed in a preset fashion, 
into a stream that can be output to the demultiplexer 26. 
The demultiplexer 26 outputs the system information 
(S), such as the video stream (V), audio stream (A) or 
the AV synchronization, forming the playback domain 
(Playltem) of the AV stream specified by the controller 
23, to the AV decoder 27, which AV decoder 27 decodes 
the video stream and the audio stream to output the 
playback video signal and the playback audio signal to 
associated terminals 32, 33, respectively. 
[0048] If fed from the terminal 24, as a user interface, 
with the information instructing random access playback 
or special playback, the controller 23 determines the re- 
adout position of the AV stream from the recording me- 
dium 100, based on the contents of the database (Clip) 
of the AV stream, to command the readout unit 28 to 
read out the AV stream. If the PlayList as selected by 
the user is to be reproduced as from a preset time point, 
the controller 23 commands the readout unit 28 to read 
out data from an !-picture having a time stamp closest 
to the specified time point. 

[0049] If the user has commanded fast forward play- 
back, the controller 23 commands the readout unit 28 
to sequentially read out l-picture data in the AV stream 
in succession based on the database (Clip) of the AV 
stream. 



[0050] The readout unit 28 reads out data of the AV 
stream from a specified random access point. The so 
read-out data is reproduced through processing by var- 
ious components on the downstream side. 
[0051 ] The case in which the user edits the AV stream 
recorded on the recording medium 100 is now ex- 
plained. If desired to specify a playback domain for the 
AV stream recorded on the recording medium 100, for 
example, if desired to create a playback route of repro- 
ducing a portion sung by a singer A from a song program 
A, and subsequently reproducing a portion sung by the 
same singer A from another song program B, the infor- 
mation pertinent to a beginning point (IN-point) and an 
end point (OUT-point) of the playback domain is input 
to the controller 23 from the terminal as a user interface. 
The controller 23 creates a database of the group (Play- 
List) of playback domains (Playltem) of the AV streams. 
[0052] When the user desired to erase a portion of the 
AV stream recorded on the recording medium 100, the 
information pertinent to the IN-point and the OUT-point 
of the erasure domain is input to the controller 23, which 
controller 23 then modifies the database of the PlayList 
so as to refer to only the needed AV streams. The con- 
troller 23 also commands the write unit 22 to erase an 
unneeded stream portion of the AV stream. 
[0053] The case in which the user desires to specify 
playback domains of an AV stream recorded on the re- 
cording medium 100 to create a new playback route, 
and to interconnect the respective playback domains in 
a seamless fashion, is now explained. In such case, the 
controller 23 creates a database of a group (PlayList) of 
the playback domains (Playltem) of the AV stream and 
undertakes to partially re-encode and remultiplex the 
video stream in the vicinity of junction points of the play- 
back domains. 

[0054] The picture information at the IN-point and that 
at the OUT-point of a playback domain are input from a 
terminal 24 to a controller 23. The controller 23 com- 
mands the readout unit 28 to read out data needed to 
reproduce the pictures at the IN-point and at the OUT- 
point. The readout unit 28 reads out data from the re- 
cording medium 100. The data so read out is output 
through the demodulating unit 29, ECC decoder 30 and 
the source depacketizer 31 to the demultiplexer 26. 
[0055] The controller 23 analyzes data input to the de- 
multiplexer 26 to determine the re-encoding method for 
the video stream (change of picture_coding_type and 
assignment of the quantity of encoding bits for re-en- 
coding) and the re-multiplexing system to send the sys- 
tem to the AV encoder 1 5 and to the multiplexer 1 6. 
[0056] The demultiplexer 26 then separates the input 
stream into the video stream (V), audio stream (A) and 
the system information (S). The video stream may be 
classed into data input to the audio decoder 27 and data 
input to the multiplexer 16. The former is data needed 
for re-encoding, and is decoded by the AV decoder 27, 
with the decoded picture being then re-encoded by the 
AV encoder 15 and thereby caused to become a video 
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stream. The latter data is data copied from an original 
stream without re-encoding. The audio stream and the 
system information are directly input to the multiplexer 
16. 

[0057] The multiplexer 16 multiplexes an input 
stream, based on the information input from the control- 
ler 23, to output a multiplexed stream, which is proc- 
essed by the ECC unit 20 and the modulation unit 21 so 
as to be sent to the write unit 22. The write unit 22 
records an AV stream on the recording medium 100 
based on the control signals supplied from the controller 
23. 

[0058] Data recorded on the recording medium 1 00 
in such recording and/or reproducing apparatus 1 (data 
reproduced from the recording medium 1 00) is repro- 
duced. Such data obtained on causing a bitstream, en- 
coded in accordance with an encoding system, such as 
the MPEG Video or MPEG Audio, and multiplexed in 
accordance with the MPEG-2 system, to become a file 
handled by a file system, and on recording the file on a 
disc, is termed an AV (audio video) stream file or a Clip 
AV stream. 

[0059] The scheme of specifying such AV stream file 
partially or entirely, arraying the specified portion for 
playback, by way of specifying the playback sequence, 
is hereinafter explained. The playback sequence delin- 
eation of specifying the AV stream file partially or entirely 
and of reproducing only the necessary portions is the 
PlayList. The PlayList is a self-contained unit in the us- 
er's perspective. The simplest structure is a unit from 
the start of recording until the end of recording. Lacking 
the editing, this unit becomes one PlayList. 
[0060] The PlayList is comprised of designation of the 
AV stream file to be reproduced and a set of a playback 
beginning point (IN-point) and a playback end point 
(OUT-point). A set of the AV stream file, playback start 
point and the playback end point is termed the Playltem. 
Thus, the PlayList is a set of PI ay Items. 
[0061] Referring to Fig. 2, the Playltem references a 
range specified by IN-point and OUT-point of a given AV 
stream. Reproducing a Playltem is tantamount to repro- 
duction of a portion of the AV stream referenced by the 
Playltem. 

[0062] While the AV stream is a bitstream multiplexed 
e.g., in the form of the MPEG-TS, the information having 
the relation of one-for-one correspondence (clip infor- 
mation) is held in a file distinct from the file in which the 
AV stream is recorded. This clip information is provided 
to facilitate reproduction and editing. This clip informa- 
tion and the AV stream is treated as one unit (object) 
and is termed a Clip. That is, the Clip is an object made 
up of the AV stream file and the ancillary information. 
[0063] Fig.3 shows the above-described relation in 
the form of a UML diagram. The structure of the AV 
stream file, Clip, Playltem and PlayList, represented by 
the UML diagram of Fig.3, renders possible the non-de- 
structive playback sequence of reproducing only option- 
al portions without modifying the AV stream file. 



[0064] The file arrangement on a recording medium 
for recording or reproducing variable information for re- 
alization of the present invention is now explained. On 
the recording medium are recorded five sorts of files, 

5 namely info.dvr, menu.thmb (mark.thmb), #####,rpls 
(##### .vpls), where ##### denotes an optional number, 
%%%%%,cpli ) where %%%%% denotes an optional 
number, and %%%%%.clpi, where %%%%% is such a 
number which causes each m2ts file to bear a one-for- 

w one correspondence with the cpli file. 

[0065] A directory /DVR is provided on a disc, so that 
a range underlying this directory is a range supervised 
by one disc respective system. It is noted that /DVR may 
be in a root directory of a disc or below a preset directory. 

is in the /DVR directory are put info.dvr, menu.thmb and 
mark.thmb files. Below /DVR are put /PLAYLIST, /CLI- 
PINF and /M2TS. The files #####.rpls and #####.vpls 
are put below /PLAYLIST, whilst %%%%%.clpi and 
%%%%%.m2ts are put in the /CLIPINF and /M2TS, re- 

20 spectively. 

[0066] A sole file, that is a file info.dvr, is put below 
/DVR. The structure of info.dvr is represented by a syn- 
tax shown in Fig. 5. The inside of a file is made up of 
blocks from one functional information to another, such 

25 that the information pertinent to the volume, the infor- 
mation pertinent to the arraying of the PlayLists and the 
information proper to the maker of the recording and/or 
reproducing apparatus 1 are stored in the DVRVolume 
(),TableOfPlayList() and in the MakersPrivateData(), re- 

30 spectively. In a leading end of a file are stated addresses 
indicating the leading ends of the respective blocks. For 
example, TableOfPlayList_Start_address indicates the 
position of beginning of TableOfPlayLists() in terms of 
the relative number of bytes in the file. 

35 [0067] The structure of DVRVolume() is represented 
by the syntax shown in Fig. 6. The version_number and 
length represent the version number of DVRVolume() 
and the length in bytes from a field directly following the 
length to the trailing end of the DVRVolumeQ. Re- 

40 sumeVolumeQ is an area in which to store the informa- 
tion pertinent to resuming, whereas UIApplnfoVoiume() 
is an area in which to store the attribute information of 
Volume. 

[0068] Fig.7 shows the syntax of UlApplnfoVolumeQ, 
45 The character_set indicates the method of encoding 
character letters encoded in the Volume_name field. 
The encoding method may be selected from ASCII, Uni- 
code and so forth. The namejength denotes the length 
of bytes of the volume (disc) name indicated in the 
so Volume_name field. The Volume_name denotes the 
name of the volume (disc). The number of bytes of the 
number of namejength from left of the field is the effec- 
tive character letter indicating the name of the volume 
(disc). 

55 [0069] In the Volume_name field, the values following 
these effective character letters are optional. The 
Volume_protect_flag is a flag indicating whether or not 
the contents in the volume may be presented to the user 
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for viewing without limitations. If 1 is set in this flag, the 
user may view the contents of the volume only if the user 
has succeeded in inputting the correct PIN number (PIN 
in Fig. 7). If the flag is set to 0, the user may view the 
contents of the volume even if the user does not input 
the PIN number. If, when the user has inserted a disc 
into a player (recording and/or reproducing apparatus 
1 ), the flag is set to 0, or the flag is set to 1 but the user 
has succeeded in correctly entering the PIN number, the 
player demonstrates a list in the PlayList in the so set 
disc. 

[0070] The foregoing is directed to playback limita- 
tions imposed on the Volume. The playback limitations 
on the respective PlayLists are irrelevant to the 
Volume_protect_flag and are indicated by the 
playback_control_flag defined in the UlApp Info Play List 
() of Fig.9, as explained subsequently. PIN is four nu- 
merical figures of from 0 to 9, each of the four numerical 
figures being encoded in accordance with the ISO/IEC 
646. If there exists a thumbnail representative of the Vol- 
ume, ref_thumbnail_index js an area in which to store 
the number used for identifying the thumbnail. The 
thumbnail in the menu and thmb files having a 
thumbnaiUndex specified by the ref_thumbnail_index 
becomes a menu thumbnail representing the Volume. 
In the present invention, thumbnails representing the 
Volume or PlayList are termed menu thumbnails. 
[0071] The rp_info_valid_flag, if 1 , indicates that next 
following rp_ref_to_PlaylistJile_name, rp_ref_to_ 
Playitemjd and rp_time_stamp are of valid values. The 
rp_ref_to_PlayList_file_name indicates that the menu 
thumbnail representing the Volume has been created 
from a picture extracted from the pictures of a preset 
PlayList, and indicates the name of the PlayList file. 
[0072] The rp_ref_to_Playltem_id indicates 
Playitemjd indicating one Playltem in the PlayList de- 
noted by rp_ref_to_PlayList_file_name, and indicates 
that the menu thumbnail representative of the Volume 
is created from the picture extracted from the picture of 
the Playltem. The rp_time_stamp indicates the presen- 
tation time stamp of a picture in the Playltem indicated 
by the rp_ref_to_PlayTimeJd, and denotes that the 
menu thumbnail representative of the Volume has been 
created from the picture. 

[0073] Fig.8 shows the syntax of TableOf PlayList(). It 
is noted that the number_of_P lay Lists indicates the 
number of PlayLists in the Volume, with the PlayList_ 
file_name specifying a filename, such as #####. rpls or 
#####vpls. In the UIApplnfoPlayList() are written vari- 
ous attributes of the PlayList, with the syntax being as 
shown in Fig.9. In the UIApplnfoPlayList(), shown in Fig. 
9, there are stored a variety of attribute information not 
directly necessary for playback of PlayList, such as 
PlayList name, recording date and time, recording time 
or erasure inhibition or allowance. Of these, 
ref_thumbnail_index is able to specify a thumbnail as a 
representative picture of the PlayList. That is, the 
thumbnail in the file menu.thmb, having the thumbnail. 



index specified by reMhumbnaii_index is the thumbnail 
representative of the PlayList. 

[0074] If rp_info_validJlag is 1 , it indicates that the 
values of the next following rp_ref_to_PlayTime_id and 

5 rp_time_stamp are valid. On the other hand, 
rp_ref_to_Playltem_id indicates the Playitemjd speci- 
fying one Playltem in the PlayList and denotes that the 
menu thumbnail representative of the PlayList is created 
from the picture extracted from the picture in the Play- 

10 item, whereas rp_time_stamp indicates the presenta- 
tion time stamp of a picture in the Playltem specified by 
the rp_ref_to_Playiterrud and denotes that the menu 
thumbnail representative of the PlayList is created from 
the picture. 

15 [0075] One file %%%%%.clpi shown in Fig.4 is creat- 
ed below /CLIPINF in association with each AV stream 
file %%%%%.m2ts. The structure of %%%%%.clpi is 
shown in Fig. 10. The inside of the file is made up of 
blocks from one functional information to another. The 

20 information pertinent to Clip is stored in CliplnfoQ, the 
information pertinent to discontinuous points of PCR 
(program clock reference) representing time base of the 
transport stream in the MPEG-2 system is stored in 
STC_lnfo(), the information pertinent to the program in 

25 the MPEG-2 system is stored in Programlnfo(), the in- 
formation pertinent to CPI (Characteristic Point Informa- 
tion) representing feature points such as random access 
start point in the AV stream is stored in CPI() and the 
mark information such as index points or commercial 

30 start and end points affixed to the Clip is stored in Clip- 
Mark. In the leading end of the file are recorded address- 
es indicating the leading ends of the respective blocks. 
[0076] Here, only the blocks relevant to thumbnails 
are explained. In the present embodiment, the operation 

35 of affixingthe thumbnails to theclip is realized by affixing 
the thumbnails to the clip marks. The information of the 
marks affixed to the clips is stored in the ClipMarkQ. Fig. 
11 shows the syntax of the ClipMark(). In the syntax, 
markjype denotes mark types, such as resume, book- 

40 mark or skip, while mark_time_stamp indicates the time 
of affixing the mark. The field pertinent to the thumbnail 
is ref_thumbnail_index. It is noted that ref_thumbnail_ 
index specifies the thumbnail number to specify the 
thumbnail in the mark.thmb file in which the thumbnail 

45 data is stored. The thumbnail specified by 
reMhumbnaiUndex becomes the mark thumbnail af- 
forded to the mark. In the present invention, the thumb- 
nail afforded to the mark is specifically termed a mark 
thumbnail. 

so [0077] The mark is mainly used to specify a highlight 
or a characteristic scene in the Clip and in the PlayList. 
The skip function of commanding skipping the playback 
as from the mark to reproduce the next Playltem is also 
realized by the mark. Fig. 12 shows that the mark is ac- 

55 corded to the Clip and to the PlayList. The features of 
the mark are now summarized. 
[0078] The mark added to the Clip specifies charac- 
teristic scenes attributable to the contents of the AV 
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stream, for example, scene change points. In reproduc- 
ing the PlayLlst, random access may be made using the 
mark in the Clip being referenced by the PlayList. In Fig. 
1 2, two marks of commercial (CM) and scene are afford- 
ed to the Clip. The commercial mark is utilized from the 
lest PlayList, whilst the scene mark is utilized from two 
PlayLists. The marks added to the PlayList, such as 
bookmarks or resume points, are mainly set by the user. 
These are bookmark and resume points in Fig. 12. 
[0079] The marks are set in the Clip or PlayList by 
adding the time stamp indicating the mark time point to 
the mark list. Mark deletion is tantamount to removing 
the time stamp of the mark from the mark list. Therefore, 
mark setting or deletion causes no changes in the AV 
stream. 

[0080] The files #####.rpls and #####.vpls, shown in 
Fig.4, are hereinafter explained. One of the files #####. 
rpls and #####.vpls is created under the /PLAYLIST for 
each PlayList. The structure of files #####. rpls and 
#####.vpls is shown in Fig. 13. The inside of the file is 
made up of blocks from one functional information to an- 
other. The information pertinent to PlayList is stored in 
PlayListf), the information of the mark afforded to the 
PlayList is stored in the PlayListMark() and the informa- 
tion proper to the maker of the recording and/or repro- 
ducing apparatus having recorded the PlayList file is 
stored in the Makers PrivateData(). In the leading end of 
the file are recorded addresses indicating the leading 
ends of blocks, such as PlayListMark_Start_address. 
So, padding_byte may be inserted ahead or at back of 
the block. However, the beginning position of the Play- 
List() is fixed and is set to e.g., 256th byte as from the 
leading end of the file. 

[0081] The contents of the block PlayList() is shown 
in Fig.1 4. First and foremost is version_number indicat- 
ing the version number of the next following information- 
The length indicates the byte length as from the field 
directly following the length to the end of the PlayList(). 
The PlayList_type indicates the sort of the PlayList and 
CPLtype denotes the type of the CPI owned by this 
PlayList. The number_of_Playltems denotes the 
number of Playltems forming the PlayList, while the 
number_of_Sub Play Items denotes the number of Play- 
items (Sub Play item) for post recording afforded to the 
PlayList. The Playltem() and the SubPlayltemQ store 
the information of the Playitem and the information of 
the Sub Playitem, respectively. 

[0082] The contents of the block Playltem() are as 
shown in Fig. 15. In the Clip_lnformation_file_name, the 
filenames of the Clip information files (files having the 
extender dpi) referenced by the Playitem are stored in 
the form of letter strings. STC_sequence_id indicates 
the domain of the time range with continuos PGR. Since 
an integrated continuous time axis may be defined in 
this domain, the start and end points of the Playitem can 
be determined uniquely. That is, the start and end points 
of the respective Playltems must be present in the same 
STC_sequence. 



[0083] lN_time denotes pts (presentation time stamp) 
on the STC_sequence of the start point of this Playitem, 
whilst OUTJime denotes the pts on the STC_sequence 
of the end point of the Playitem. The connection_con- 
5 dition is the information indicating how this Playitem is 
coupled to the next Playitem and denotes the condition 
as to whether or not the neighboring Playitems can be 
reproduced seamlessly. 

[0084] BridgeSequencelnfo() stores the information 
10 pertinent to a bitstream (bridge sequence) generated 
when skipping to a bitstream different from the bitstream 
to be inherently reproduced at a junction of the Play- 
items and reproducing this different bitstream to realize 
the function of reproducing the neighboring Playitems 
is seamlessly. The program_n umber denotes a set of el- 
ementary streams, such as video or audio streams, de- 
fined in MPEG system, that is a channel in the television 
broadcast. 

[0085] The above is the gist of the data structure form- 

20 jng the PlayList and the Playitem. With such data struc- 
ture, it is possible to form the PlayList by an array of 
Playitems in the AV stream which are desired to be re- 
produced and which are specified by sets of IN and OUT 
points, while it is also possible to supervise a set of play- 

25 back unit recognized by the user. 

[0086] In the embodiment of the present invention, the 
operation of affixing a thumbnail at an optional time point 
on the PlayList is realized by affixing a thumbnail to the 
PlayList mark. The mark information affixed to the Play- 

30 List is stored in the PlayListMark(). Fig. 16 shows the 
syntax of the PlayListMarkQ. The mark_type in the syn- 
tax denotes the type of the mark, such as resume, book- 
mark and skip, while mark_time_stamp denotes the 
time the mark is assigned. The ref_thumbnail_index 

35 specifies the thumbnail number to specify the thumbnail 
in the mark.thmb file in which thumbnail data is stored. 
The thumbnail specified by ref_thumbnaiLindex be- 
comes the mark thumbnail corresponding to the mark. 
[0087] The thumbnail is explained in detail. The 

40 thumbnail denotes a still picture ancillary to Volumes, 
PlayList and Clip. There are two sorts of the thumbnail, 
one being a thumbnail representing the contents. This 
thumbnail is mainly used in a menu picture in order to 
permit the user to select the picture desired to be viewed 

45 using a cursor. The other is a picture representing a 
scene pointed by the mark. 

[0088] The Volume and each PlayList need to be able 
to have respective representative pictures. The repre- 
sentative picture of the Volume is presupposed to be 

50 used for such a case wherein a still picture indicating 
the disc contents are displayed first. The representative 
picture of the PlayList is presupposed to be used as a 
still picture for demonstrating the PlayList contents in 
the menu picture used in selecting the PlayList. 

55 [0089] The simplest method of realizing the repre- 
sentative picture of the PlayList is to render the first pic- 
ture of the PlayList a thumbnail. However, the leading 
picture at the playback time 0 is not necessarily optimum 
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in representing the contents. So, an optional picture is 
adapted for being determined as a thumbnail on the 
PlayList. The above two thumbnails are termed menu 
thumbnails. The menu thumbnails are demonstrated 
frequently and hence need to be able to be read out 
speedily from the disc. For satisfying the need, all menu 
thumbnails are stored in one file for the sake of efficien- 
cy. The menu thumbnails are not necessarily the pic- 
tures extracted from the moving pictures in the volume, 
but may be pictures captured from the personal compu- 
ter or from a digital still camera. 
[0090] On the other hand, it is necessary that plural 
marks can be entered in the Clip and PlayList and that, 
for grasping the contents of the mark positions, the mark 
point pictures need to be able to be viewed easily. The 
pictures indicating these mark points are termed mark 
thumbnails. So, the thumbnails are derived mainly form 
the extracted markpoint pictures, rather than the pic- 
tures captured from outside, as shown in Figs. 18 and 
19. In distinction from the menu thumbnails, the mark 
thumbnails are used e.g., in a submenu used for repre- 
senting details of the PlayList, and hence need not be 
read out in short access time. 

[0091] So, there is no problem if the operation of the 
player opening a file to read out a portion of the file each 
time a thumbnail is necessary is somewhat time-con- 
suming. For decreasing the number of files present in 
the volume, all mark thumbnails are desirably stored in 
one file. Although the Play List may have one menu 
thumbnail and plural mark thumbnails, the Clip need not 
be selected directly by the user (it is usually specified 
through PlayList) so that it does not have the menu 
thumbnail. Fig.20 shows the above-described relation. 
[0092] Since the thumbnails are added and deleted 
every so often, the operations of addition and partial de- 
letion need to be executed readily and speedily. For this 
reason, the ThumbnailQ has a block structure. Picture 
data are divided into plural portions each of which is 
stored in one tn_block(). One picture data is stored in 
continuous tn_block(). There may exist tn_block() not in 
used in the string of the tn_block(). The byte length of 
one thumbnail picture is variable. 
[0093] Fig.21 shows a syntax of the file which stores 
thumbnail data. In this file, there exists only one Thumb- 
nail{). Fig.22 shows the syntax of thumbnail(). The 
version_number denotes four character letters indicat- 
ing the version number of this thumbnail(). The length 
is a 32-bit unsigned integer indicating the number of 
bytes of thumbnail(), whilst tu_blocks_start_address is 
a 32-bit unsigned integer indicating the start byte ad- 
dress as from the leading end of thumbnail(). 
[0094] The number_oMhumbnails is a 16-bit un- 
signed integer indicating the number of thumbnail pic- 
tures stored in this thumbnail (). The tu_blocks_start_ 
address is a 16-bit unsigned integer representing the 
size of 1 tn_block() in kilobytes. For example, the 
tu_blocks_start_address being 1 indicates that the size 
of one tn_block() is 1024 bytes. The number_of_ 



tu_blocks is a 16-bit unsigned integer indicating the 
number of the tn_block() in this thumbnail(). 
[0095] The thumbnail_index is a 1 6-bit unsigned inte- 
ger indicating an index number of a thumbnail repre- 

5 sented by the thumbnail information for one for-loop be- 
ginning from this thumbnail_index field. The value of 
OxFFF must not be used for thumbnaiUndex. The 
thumbnailjndex is referenced by reMhumbnail_index. 
The thumbnail_picture_format is an 8-bit unsigned inte- 

10 ger indicating a picture format of a thumbnail picture and 
assumes a value shown in Fig.23. The DCF (Design 
Rule for camera File System) and the PNG (Portable 
Network Graphics) are allowed only within "menu, 
thmb". That is, the mark thumbnail must assume a value 

15 "0x00" (MPEG-2 Video l-picture). 

[0096] The picture_data_size is a 32-bit unsigned in- 
teger indicating the byte length of an encoded thumbnail 
picture, while the start_tn_block_n umber is a 1 6-bit un- 
signed integer indicating the tn_block_number of 

20 tn ^blocks with which begins data of a thumbnail picture. 
The leading end of thumbnail picture data must coincide 
with the leading end of tb_block. The start_tn_block_ 
number begins with 0 and is relevant to the number of 
a variable k in the tn_block for-loop. The x_picture_ 

25 length is a 1 6-bit unsigned integer indicating the number 
of pixels in the horizontal direction of the thumbnail pic- 
ture. 

[0097] The y_picture Jength is a 1 6-bit unsigned inte- 
ger indicating the number of pixels in the vertical direc- 

30 tion of the thumbnail picture. The tu_block is an area in 
which to store picture data. All of tn_blocks() in the 
thumbnail() must be of the same size prescribed in 
tn_block_size. Figs.24A and 24B schematically indicate 
the manner in which the picture data are stored in the 

35 tn_block. As shown in Figs.24A and 24B, each picture 
data begins at the leading end of tn_block(). If a picture 
data is of a size exceeding one in_block(), it is stored 
by exploiting the next following tn_block(). 
[0098] Although picture data is of variable length, one 

40 in_block() is of a fixed size. By converting the variable 
length data into fixed length data, addition of new picture 
data or deletion of picture data can be dealt with on the 
block basis, thus simplifying address management. 
[0099] An alternative proposal of the directory file 

45 structure and the syntax for thumbnail recording is here- 
inafter explained. An alternative proposal of the directo- 
ry file structure shown in Fig.4 is shown in Fig.25. In Fig. 
4, there is only one file for recording a mark thumbnail, 
namely mark.thmb, below /DVR. In Fig.25, plural files 

50 xxxxx.thmb are provided in a one-for-one correspond- 
ence for each PlayList and Clip (xxxxx is a filename, ex- 
cluding an extender, representing an associated PlayL- 
ist or Clip). 

[0100] For example, the mark thumbnail picture, 
55 owned by the PlayList represented by #####. rpls, is 
stored in #####.thmb in the same directory. Similarly, 
the picture of the mark thumbnail owned by the Clip rep- 
resented by%%%%%.clpi, is stored in %%%%%.thmb 
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in the same directory. The mark thumbnail of the Play- 
List clip is subjected to the constraint that it is stored in 
the .thumb file having the same name portion except the 
file extender, with the syntax in the xxxxx.thmb being the 
same as menu.thmb or mark.thmb (Fig.21), as shown 5 
in Fig.26. 

[0101 J As for the menu thumbnail of the PlayList, the 
ref_thumbnail_index specifying a thumbnail is placed in 
Figs. 8 and 9 in UIApplnfoPlayList() in TableOf PlayLists 
() of an Info.dvrfile. Alternatively, the UlApp Info PlayList 10 
0 may be placed not in Info.dvr but in each PlayList file, 
in such case, the UIApplnfoPlayList() is moved from Ta- 
bleOf PlayListsQ to PlayListQ in the PlayList file. So, the 
syntax shown in Fig.8 is as the syntax shown in Fig. 27, 
whilst the syntax shown in Fig. 1 4 is as the syntax shown is 
in Fig.28. The UlApptnfoPlayListQ in Fig.28 is as the 
syntax shown in Fig.29, in which is inserted ref_ 
thumbnaiLindex. 

[0102] Thumbnail recording is possible with the 
above-described file and the syntax. 20 
[0103] Referring to the flowchart of Fig. 30, creation of 
a mark thumbnail is explained. At step S1 , the user se- 
lects the PlayList desired to be reproduced. At step S2, 
an AV stream starts to be reproduced, based on the se- 
lected PlayList. At step S3, the user views the AV stream 25 
being reproduced to search a scene desired to be 
marked. If the user has succeeded in searching the 
scene to be marked, the or she acts on a mark button 
of a remote controller, not shown, annexed to the re- 
cording and/or reproducing apparatus 1 . This operation 30 
may be made during playback or during pause. 
[01 04] If a mark button is acted on by the user at step 
S3, the mark position is determined at step S4 as a 
processing corresponding to user command. If the mark 
position is set, a picture which is to be a thumbnail pic- 35 
ture is selected at step S5. At a time point when the mark 
position is specified, the controller 23 (Fig. 1) may inquire 
of the user whether or not the thumbnail is to be formed 
and, if the thumbnail is to be formed, the controller 23 
may furnish a candidate thumbnail picture to the user 40 
by e.g., intermittent feed to permit the user to select the 
thumbnail picture. 

[0105] If the picture which is to be the thumbnail pic- 
ture has been selected by the user or by the controller 
23, the controller 23 captures a picture from the AV en- 45 
coder 15 to transfer the picture to a RAM (random ac- 
cess memory), not shown, of the controller 23. At step 
S7, it is verified whether or not the picture is to be com- 
pressed. If the picture is verified to be compressed, the 
controller 23 transfers to step S8 to effect picture com- so 
pression to return data to the RAM. if it is verified at step 
S7 that the picture is not to be compressed, the control- 
ler 23 skips the processing of step S8 to transfer to step 
S9. 

[0106] At step S9, the controller 23 creates the header 55 
information, such as thumbnail encoding system, 
thumbnaiUd for identifying a thumbnail, size (number 
fbytes) and the thumbnail header constituted by the 
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number of pixels in the X- and Y-directions, to divide the 
picture data into tu_block() units. At step S10, picture 
data split into the header information and the tn_block() 
is transferred to the ECC unit 20 through write unit 22 
so as to be written as mark.thmb file on the recording 
medium 100. 

[0107] Referring to theflowchart of Fig.31 , the method 
for creating a menu thumbnail is explained. Since the 
menu thumbnail is assigned to the Volume or to each 
PlayList, the controller 23 at step S21 allows the user to 
select the object to which to afford the menu thumbnail. 
At step s22, the controller 23 allows the user to select 
whether a thumbnail is to be captured from outside or 
from a scene in the PlayList. 

[01 08] If it is verified at step S22 that the thumbnail is 
to be captured from outside, the controller 23 transfers 
to step S23 where picture data is captured from an input 
terminal specified by the user or from a file. When the 
capturing is finished, the controller 23 proceeds to step 
S26. 

[01 09] If it is verified at step S22 that the thumbnail is 
not to be captured from outside, the controller 23 trans- 
fers to step S24 where the controller 23 first allows the 
user to select from which PlayList the thumbnail picture 
is to be captured. The controller 23 then proceeds to 
reproduce the selected PlayList. At step S25, the con- 
troller 23 allows the user to select the scene desired to 
be rendered a thumbnail. 

[0110] If the thumbnail picture has been selected at 
step S23 or step S25, the controller 23 proceeds to step 
S26. The processing as from step S26 is similar to the 
processing as from step S5 of the flowchart of Fig. 30 
and hence is not explained specifically. 
[0111] By affording the thumbnail picture to the Play- 
List as described above, these thumbnail pictures are 
easier to supervise, thus facilitating, the processing of 
selecting the AV stream in reproducing the recorded AV 
stream. 

[0112] Fig.32 shows another embodiment of the di- 
rectory and the file to be created on the disc. The "menu, 
tidx" and "menu.tdat M store the information of a repre- 
sentative picture for each menu thumbnail, that is each 
picture and each PlayList representing the Volume. The 
header information of the totality of the menu thumbnails 
is managed as a compact menu.tidx. The picture data 
of the totality of the menu thumbnails is managed as a 
compact menu.tdat. 

[01 1 3] The "mark.tidx" and "mark.tdat" store the infor- 
mation pertinent to a mark thumbnail, that is a picture 
specified by a mark point. The header information of the 
totality of the mark thumbnails is managed in a compact 
mark.tidx, while picture data of the totality of the mark 
thumbnails is managed in a compact mark.tdat. 
[01 14] That is, in the present fie structure, the thumb- 
nail header information and the thumbnail picture data 
are managed in separate files. The reason is relevant 
to the file management method undertaken by the file 
system. That is, since the file of the thumbnail header 
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information is the relatively crucial information, the file 
system manages the data by dual writing on the disc. 
[01 1 5] The reason of doing this is to deal successfully 
with the case of one of the data being lost due to scratch- 
es on the disc, and has the meaning of data backup. 
The picture data file is not written in duplex because it 
is of relatively large volume such that the capacity re- 
quired on the disc is increased due to dual writing. 
[01 1 6] The encoding method, sampling structure and 
scanning structure for picture data stored in the "menu, 
tdat" and "mark.tdat" can be changed from picture to pic- 
ture. However, the recording and/or reproducing appa- 
ratus 1 can be simpler in structure if the totality of the 
picture data are the same. For example, the JFIF (JPEG 
file interchange format), component signals and pro- 
gressive scan format are used. 
[0117] The syntax and the semantics of these four 
files are explained. The "menu.tidx" and "mark.tdat" are 
of the same syntax structure. Fig. 33 shows the syntax 
structure of "menu.tidx" and "mark.tdaf . The version, 
number is the four numerical figures indicating the ver- 
sion number of the thumbnail header information file. 
[01 1 8] The length is the number of bytes as a from the 
byte directly following the length field to the last byte of 
the menu.tidx/mark.tidx. In case of menu.tidx, the 
number_of_thumbnails is the number of thumbnails 
stored in the menu.tdat and, in case of the mark.tidx, it 
is the number of the thumbnail pictures stored in the 
mark.tdat. 

[0119] In the cases of menu.tidx and mark.tidx, 
tn_block_size indicates the size of one tn_block in 
menu.tdat and the size of of one tn_block in mark.tdat, 
respectively. This size is based on 1 024 bytes as a unit. 
For example, tn_block_size =1 indicates that the size of 
one tn_block is 1 024 bytes. One thumbnail picture must 
be stored in one tn_block. 

[0120] In the cases of menu.tidx and mark.tidx, the 
number_of_tn_b locks indicates the number of 
tn_blocks in the menu.tdat and that in the mark.tdat, re- 
spectively. The thumbnaiL'mdex indicates the index 
number of the thumbnail information next following the 
thumbnail_index field. As the thumbnaiLindex, the val- 
ue of OxFFFF must be used. 

[0121] In the case of menu.tidx, the thumbnaiLindex 
is referenced by reMhumbnaiUndex in U I App Info Vol- 
ume and in UlApplnfoPlayList. In the case of mark.tidx, 
the thumbnaiLindex is referenced by reMhumbnailJn- 
dex in Playl_istMark() and in ClipMark(). 
[0122] In the case of menu.tidx, ref_to_block_id indi- 
cates one tn_block in menu.tdat, with the tn_block stor- 
ing picture data indicated by thumbnaiLindex. The val- 
ue of ref_to_block_id references the value of 
tn_blockJd in the menu.tdat syntax. 
[0123] In the case of mark.tidx, reMo_block_id indi- 
cates one tn_block in mark.tdat, with the tn_block stor- 
ing picture data indicated by thumbnaiLindex. The val- 
ue of reLto_b!ock_id references the value of 
tn_block_id in the menu.tdat syntax. 
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[0124] The picture_byte_size indicates the data 
length of one encoded thumbnail picture pointed by 
thumbnaiLindex in bytes. The picture_byte_size must 
be less than the value of 1024*tn_block_size. That is, 
5 the encoding by the recording and/or reproducing appa- 
ratus 1 must be such that the data length of one encoded 
thumbnail picture will be not larger than 1 024 ± value of 
tn_block_size. 

[0125] The horizontal_picture_size indicates the 
10 number of pixels in the horizontal direction of the encod- 
ed thumbnail picture pointed by thumbnaiLindex. The 
vertical _picture_size indicates the number of pixels in 
the vertical direction of the encoded thumbnail picture 
pointed by thumbnaiLindex. The display_aspect_ratio 
15 shows a display aspect ratio of the encoded thumbnail 
picture pointed by thumbnaiLindex. The meaning of the 
values is shown in Fig. 34. 

[0126] The color_space indicates the format when 
converting component signals of Y, Cb and Cr into com- 
20 ponent signals of R, G and B. The meaning of the values 
is shown in Fig.35. 

[0127] The "menu.tdat" and the "mark.tdat" are of the 
same syntax structure. Fig.36 shows the syntax struc- 
ture of "menu.tdat" and the "mark.tdat". The tn_block is 

25 an area in which to store one encoded thumbnail picture. 
The byte length of one thumbnail picture is not larger 
than the size of one in_block. The first byte of one picture 
data must coincide with the first byte of tn_block. 
[0128] In the cases of "menu.tdat" and "mark.tdat", 

30 the size of one tnjalock is indicated by the tn_block_size 
in menu.tdat and by the tn_block_size in mark.tdat, re- 
spectively. Each tn_block is differentiated by the value 
of tn_block_id when the tn_block appears in the syntax 
for-loop. In the cases of "menu.tidx" and "mark.tidx", 

35 tn_block_id is referenced by ref_to_tn_blockJd in the 
menu.tidx and by ref_to_tn_block_id in the mark.tidx, re- 
spectively. 

[0129] Since the thumbnails are added and deleted 
every so often, the operations of addition and partial de- 

40 letion need to be executed readily and speedily. For this 
reason,"menu.tdat" and "mark.tdat" are of a block struc- 
ture. One picture data is stored in one tn_block(). 
[0130] A tn_block not in use may be present in the 
tn_block string of "menu.tdat" and "mark.tdat". For ex- 

45 ample, if, in deleting a preset thumbnail, the 
thumbnaiLindex entered in a header information file of 
the thumbnail is erased, without changing the picture 
data file of the thumbnail, tn_blocks not in use are pro- 
duced in the tn_block string. 

so [0131] Fig. 37 schematically shows the manner in 
which thumbnail picture data are stored in the tn_block. 
As shown in Fig37, the byte length of one thumbnail pic- 
ture is not larger than the size of one tn_block. A 
tn_block not in use may be present on one tn_block. 

55 [0132] In the above embodiment, it is assumed that 
menu_tbat and mark.tbat files are recorded as they are 
divided into two. Alternatively, each file may be con- 
ceived as being one block and a first block of menu.tbat 
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data and a second block of mark.tbat data may be re- 
corded as being one compact file. In recording in one 
compact file, Sequencelnfo, CPI and Clipmark may be 
recorded as separate blocks. 

[0133] The above-described sequence of operations 
may be executed not only by hardware but also by soft- 
ware. If the sequence of operations is to be executed by 
software, it is installed from a recording medium to a 
computer in the dedicated hardware of which the pro- 
gram forming the software is installed or to a general- 
purpose personal computer of Fig. 38 capable of execut- 
ing various functions based on a variety of programs in- 
stalled therein. 

[0134] In the personal computer, shown in Fig. 38, a 
CPU (central processing unit) 201 executes a variety of 
operations in accordance with programs stored in a 
ROM (read-only memory) 202 or programs loaded from 
a memory 208 to a RAM (random-access memory) 203. 
There is also stored in the RAM 203 data required in 
executing various processing operations. 
[0135] The CPU 201 , ROM 202 and the RAM 203 are 
interconnected over a bus 204. An input/output interface 
205 is also coupled to the bus 204. 
[0136] To the input/output interface 205 are coupled 
an input unit 206, comprised of a keyboard and a mouse, 
a display comprised of CRT and an LCD, an output unit 
207, comprised of a speaker, a memory 208, comprised 
of a hard disc and a communication unit 209 comprised 
of a modem and a terminal adapter. The communication 
unit 209 executes communication processing over a 
network. 

[0137] To the input/output interface 205, there are al- 
so connected a drive 210, a magnetic disc 221 , an op- 
tical disc 222, a magneto-optical disc 223 or a semicon- 
ductor memory 224. The program read out therefrom is 
installed as necessary on the memory 208. 
[0138] The recording medium is constituted not only 
by a package medium distributed for furnishing the pro- 
gram to the user, in addition to a computer, such as a 
magnetic disc 221 carrying the program thereon, inclu- 
sive of a floppy disc, an optical disc 222 inclusive of a 
CD-ROM (Compact Disc-Read-Only memory) or a DVD 
(Digital Versatile Disc), a magneto-optical disc, inclusive 
of a Mini-Disc, or a semiconductor memory 224, but also 
by a hard disc, inclusive of a ROM 202 carrying a pro- 
gram and a memory 208, furnished to the user as it is 
built-in in a computer, as shown in Fig. 38. 
[0139] In the present specification, the steps of the 
program furnished by the medium include not only the 
chronological processing in accordance with the se- 
quence indicated, but also the processing not chrono- 
logically but in parallel or separately. 
[0140] Additionally, in the specification, the system 
means an entire apparatus comprised of plural compo- 
nent devices. 
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Industrial Applicability 

[0141] In the recording method and apparatus and in 
the first program, according to the present invention, a 

5 thumbnail picture representing the moving picture data 
is generated from input moving picture data as first 
thumbnail data. Form the moving picture data, a thumb- 
nail picture of a picture extracted as being a character- 
istic picture, or a thumbnail picture of a picture specified 

10 by a user, is generated as second thumbnail data, and 
the first and second thumbnail data are recorded as re- 
spective independent groups on the recording medium. 
So, the user is allowed to select recorded data readily 
using the recorded picture data. 

is [0142] In the reproducing method and apparatus and 
in the second program, according to the present inven- 
tion, if playback of an AV stream is commanded, man- 
agement data supervising picture data of a thumbnail 
picture representing the contents of the AV stream is 

20 read out, and the picture data is read out based on the 
management data read out by readout means. From the 
AV stream, picture data of a thumbnail picture of a pic- 
ture extracted from it as a characteristic, picture or pic- 
ture data of a thumbnail picture of a picture specified by 

25 a user, and management data supervising the picture 
data, are read out, and the picture data are read out 
based on the read-out management data. So, the user 
is allowed to select recorded data readily using the read- 
out management data. 

30 

Claims 

1 . A recording apparatus comprising: 

35 

means for generating, from input moving pic- 
ture data, a thumbnail picture representing said 
moving picture data, as first thumbnail data, 
and for generating, from said moving picture 
40 data, a thumbnail picture of a picture extracted 

as a characteristic picture, or a thumbnail pic- 
ture of a picture specified by a user, as a second 
thumbnail data; and 

means for recording said first thumbnail data 
45 and the second thumbnail data, generated by 

said generating means as independent groups 
on a recording medium. 

2. The recording apparatus according to claim 1 
so wherein said generating means generates said first 

and second thumbnail data as respective independ- 
ent files. 

3. The recording apparatus according to claim 1 
55 wherein said generating means generates said first 

and second thumbnail data as respective independ- 
ent data blocks and as one file. 
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4. The recording apparatus according to claim 1 
wherein said generating means generates first 
management data supervising said first thumbnail 
data with a number corresponding to the first 
thumbnail data, said generating means also gener- 
ating second management data supervising said 
second thumbnail data with a number correspond- 
ing to the second thumbnail data; 

said recording means recording said first and 
second management data in said recording medi- 
um. 

5. The recording apparatus according to claim 4 
wherein said first and second management data in- 
clude data specifying the format of the picture data 
of the thumbnail data being supervised. 

6. The recording apparatus according to claim 4 
wherein said recording means records picture data 
of said thumbnail picture contained in said first or 
second management data in terms of a block of a 
preset size as a unit. 

7. The recording apparatus according to claim 1 
wherein said recording means records the informa- 
tion representing the referencing destination of said 
first thumbnail data as a separate file on said re- 
cording medium. 

8. The recording apparatus according to claim 1 
wherein said recording means further records the 
information indicating the referencing destination of 
said thumbnail picture contained in said second 
thumbnail data. 

9. A recording method comprising: 

a generating step of generating, from input 
moving picture data, a thumbnail picture repre- 
senting said moving picture data, as first 
thumbnail data, and for generating, from said 
moving picture data, a thumbnail picture of a 
picture extracted as a characteristic picture, or 
a thumbnail picture of a picture specified by a 
user, as a second thumbnail data; and 
a recording controlling step of exercising con- 
trol for recording said first thumbnail data and 
the second thumbnail data, generated by said 
generating step as independent groups on a re- 
cording medium. 

10. A recording medium having recorded thereon a 
computer-readable program, said computer-reada- 
ble program comprising: 

a generating step of generating, from input 
moving picture data, a thumbnail picture repre- 
senting said moving picture data, as first 



thumbnail data, and for generating, from said 
moving picture data, a thumbnail picture of a 
picture extracted as a characteristic picture, or 
a thumbnail picture of a picture specified by a 

5 user, as a second thumbnail data; and 

a recording controlling step of exercising con- 
trol for recording said first thumbnail data and 
the second thumbnail data, generated by said 
generating step as independent groups on a re- 

10 cording medium. 

1 1 . A program for having a computer execute 

a generating step of generating, from input 
moving picture data, a thumbnail picture represent - 

15 ing said moving picture data, as first thumbnail data, 
and for generating, from said moving picture data, 
a thumbnail picture of a picture extracted as a char- 
acteristic picture, or a thumbnail picture of a picture 
specified by a user, as a second thumbnail data; 

20 and 

a recording controlling step of recording said 
first thumbnail data and the second thumbnail data, 
generated by said generating step as independent 
groups on a recording medium. 

25 

12. A reproducing apparatus comprising: 

first readout means for reading out manage- 
ment data supervising picture data of a thumb- 
30 nail picture representing the contents of said 

picture data in case playback of said picture da- 
ta is commanded; 

second readout means for reading out said pic- 
ture data based on said management data read 

35 out by said readout means; 

third readout means forreading out picture data 
of a thumbnail picture of a picture extracted as 
a characteristic picture from said picture data 
or picture data of a thumbnail picture of a pic- 

40 ture specified by a user, and management data 

supervising said picture data; and 
fourth readout means for reading out said pic- 
ture data based on said management data read 
out from said third readout means. 

45 

13. A reproducing method comprising: 

a first readout controlling step of reading out 
management data supervising picture data of 

so a thumbnail picture representing the contents 

of said picture data in case playback of said pic- 
ture data is commanded; 
a second readout controlling step of controlling 
the readout of said picture data based on said 

55 management data readout-controlled by the 

processing of said first readout control means; 
a third readout controlling step of reading out 
picture data of a thumbnail picture of a picture 
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extracted as a characteristic picture from said 
picture data or picture data of a thumbnail pic- 
ture of a picture specified by a user, and man- 
agement data supervising said picture data; 
and 

a fourth readout controlling step of reading out 
said picture data based on said management 
data readout-controlled by the processing of 
said third readout control step. 

14. A recording medium having recorded thereon a 
computer- readable program, said computer- reada- 
ble program comprising: 

a first readout controlling step of controlling the 
readout of management data supervising pic- 
ture data of a thumbnail picture representing 
the contents of said picture data in case play- 
back of said picture data is commanded; 
a second readout controlling step of controlling 
the readout of said picture data based on said 
management data readout-controlled by the 
processing of said first readout control means; 
a third readout controlling step of reading out 
picture data of a thumbnail picture of a picture 
extracted as a characteristic picture from said 
picture data or picture data of a thumbnail pic- 
ture, of a picture specified by a user, and man- 
agement data supervising said picture data; 
and 

a fourth readout controlling step of reading out 
said picture data based on said management 
data readout-controlled by the processing of 
said third readout controlling step. 

15. A program for having a computer execute 

a first readout controlling step of controlling 
the readout of management data supervising pic- 
ture data of a thumbnail picture representing the 
contents of said picture data in case playback of 
said picture data is commanded; 

a second readout controlling step of control- 
ling the readout of said picture data based on said 
management data readout-controlled by the 
processing of said first readout control means; 

a third readout controlling step of reading out 
picture data of a thumbnail picture of a picture ex- 
tracted as a characteristic picture from said picture 
data or picture data of a thumbnail picture of a pic- 
ture specified by a user, and management data su- 
pervising said picture data; and 

a fourth readout controlling step of reading out 
said picture data based on said management data 
readout-controlled by the processing of said third 
readout control step. 

16. A recording medium having recorded thereon first 
data comprised of picture data, thumbnail picture 



data of a thumbnail picture extracted from said pic- 
ture data, indicating the contents of the picture data, 
and management data for supervising data of said 
thumbnail picture, and second data comprised of 
5 thumbnail picture data of a thumbnail picture of a 
picture extracted as a characteristic picture from 
said first stated picture data or picture data of a 
thumbnail picture of a picture specified by the user 
and management data supervising the picture data. 

10 
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